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The  linear  quadratic  optimal  control  method 
is  used  today  to  solve  many  complex  systems 
problems.  As  system  complexity  increases,  and 
as  linear  quadratic  optimal  control  is  used  in 
more  demanding  situations,  the  extension  of  the 
design  methodology  to  cover  system  failures, 
robustness  and  reliability  is  of  crucial  impor- 
tance. This  paper  documents  the  progress  toward 
a theory  which  incorporates  reliability  in  the 
performance  index;  a linear  quadratic  control 
problem  is  formulated  which  accounts  for  system 
effectiveness  and  gives  an  offline  procedure  for 
comparing  two  linear  quadratic  control  systems 
on  the  basis  of  both  reliability  and  performance. 


1 . Introduction 

This  paper  deals  with  the  formulation  of 
and  a suboptimal  solution  to  an  optimal  control 
problem  over  a system  with  randomly  varying 
structure.  The  first  objective  of  this  research 
is  to  create  a procedure  which  can  be  used  in 
offline  design  tradeoff  studies  of  various 
system  designs  using  different  numbers  of  actua- 
tors with  varying  failure  rates  and  which  judges 
a given  design  on  the  basis  of  both  performance 
euid  reliability.  Secondly,  we  wish  to  establish 
a dividing  line  between  reliable  and  unreliable 
systems. 

Whenever  one  considers  the  optimal  control 
of  systems  with  varying  structure,  the  con^lexlty 
of  the  solution,  even  when  it  exists,  is  enormous. 
In  any  design  technique  involving  these  systems, 
there  is  a tradeoff  between  the  complexity  of  the 
technique  and  the  2unount  of  information  it  pro- 
duces. The  approach  taken  in  this  research  is  to 
simplify  the  procedure  as  much  as  possible  while 
retaining  its  ability  to  weigh  system  designs  on 
the  basis  of  both  performance  and  reliability. 

The  solution  presented  herein  serves  a 
second  purpose:  It  allows  the  researcher  to 
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study  the  effect  of  various  actuator  configurations 
and  failure  modes  on  system  performance  and  to  gain 
a better  understanding  of  the  relationship  between 
rell^d^ility  and  control  theory.  This  is  perhaps 
the  more  important  aspect  of  this  research. 

Previously,  several  authors  have  studied  the 
optimal  control  of  systems  with  randomly  varying 
structure.  Most  notable  among  these  is  Wonham  in 
[71 , where  he  develops  a solution  to  the  linear 
regulator  problem  with  randomly  jumping  parameters. 
This  solution,  however,  assumes  a^  priori  that  the 
controller  has  perfect  information  about  the  present 
state  of  the  random  process.  Thus,  the  control 
depends  on  the  random  parameter.  In  reality,  one 
could  not  expect  to  have  perfect  information,  how- 
ever, and  a noisy  observation  of  the  random  process 
leads  to  the  dual  control  problem.  Wonham  made  a 
major  contribution,  however,  in  deriving  the  form 
of  the  coupled  Riccati  equations  to  which  this  type 
of  problem  leads.  Sworder  has  used  this  informa- 
tion in  some  specific  cases  [4,5],  as  have  Ratner 
and  Luenberger  [3] . Bar-Shalom  and  Slvan  [1]  pre- 
sented a variation  of  Wonham 's  work  by  using  the 
optimal  open  loop  controller  at  each  step  of  a 
discrete-time  linear  system  with  random  structure. 
Winner  [6]  developed  a suboptimal  control  scheme, 
which  allowed  for  imperfect  observation  of  the 
random  parameters,  known  as  multiple-model  adaptive 
control.  In  this  method,  the  parameters  could  only 
take  a discrete  set  of  values,  a cause  of  recent 
disfavor,  as  MMAC  does  not  work  well  when  the 
parameters  vary  continuously  and  are  approximated 
by  the  mathematics.  Similar  work  has  been  done 
by  Pierce  euid  Sworder  in  [2] . 


2 . Motivation 

To  Illustrate  the  dual  control  problem,  and  to 
motivate  this  work,  consider  the  following  simple 
example.  Let  the  system  be  one  dimensional  with 
one  control  variable. 

x(t+l)  » ax(t)  + bjjU(t)  (1) 

The  value  of  the  control  multiplier  (b)^)  is  a 
random  variable  which  takes  on  one  of  two  discrete 
values  at  each  time  t. 

b if  k - 0 

bvCt)  - (2) 

1/b  if  k - 1 


Sot  p jbilo  rdit»»i,w  , 
distribution  unliniitod. 


k . 
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The  randoa  process  Is  governed  by  the  Harkov  chain 
represented  by 


V*(x(t),k-i,t)  - x''(t)Sj^(t) 
Thus, 


(11) 


lt{t+l)  - PH(t) 


where 


¥(t)  e 


(3) 


(4) 


u(t)  (12) 

l¥o(t+l)abSo(t+l)  + ¥j^(t+l)a/bSi(t+l)lx(t) 
r + ¥p(t+l)b2So(t+l)  + ¥j^(t+l)l/b2Sj^(t+l) 


p ■ 

Pll 

P12 

e r2=‘2 

(5) 

where 

¥(t+l)  - N !!!:'!] 

= P¥(t) 

(13) 

P21 

P22' 

- l¥i(t+l)J 

At  any  given  time  t,  the  following  sequence  of 
events  occurs: 

I)  x(t)  is  observed  exactly,  b]((t-l)  is 
coiiq>uted,  2md  k(t)  is  set  to  0 or  1 
depending  on  by(t-l),  where  k(t)  is  the 
variable  representing  the  Harkov  c)iain! 

II)  b]^  may  c)uinge  values  to  b)c(t); 

III)  u(t)  is  applied. 

For  any  given  saiq>le  path,  the  performance  index 
is  given  by 
T 

J “ X(qx^(t)  + ru^(t))  (6) 

t-0 

where  {0,1,...,t}  is  the  time  set  over  which  the 
system  is  to  be  controlled.  The  objective  of  the 
control  problem  is  to  minimize  the  expected  cost- 
to-go  at  t,  given  by 

V(x(t)  ,k(t)  ,u(t)  ,t) 

T 

- Ei  I (qx2(T)  + ru2(T))  |x(t))  (7) 

T-t 

where  the  expectation  is  taken  over  all  possible 
sample  paths  of  k(T),  t<  t^T.  Using  dynamic 
prograsmlng , we  wish  to  minimize 

V(x(t),k(t),u(t),t) 

- E(qx^(t)  + ru2(t)  (8) 

+ V*(ax(t)+b^(t)u(t),k(t+l),t+l)|x(t)) 

where  V* (• ,k(t+l) ,t+l)  represents  the  minimum  cost- 
to-go,  given  k(t-i-l)  at  time  t+1. 

This  minimization  can  t>e  carried  out  because 
x(t)  is  known  exactly  at  time  t.  Given  x(t). 


SQ(t)  and  S]^(t)  are  propagated  Isackward  in  time  by 
the  following  equations; 


Assuming  k-0  at  time  t,  then  ¥(t+l) 
So(t) 


[Pll  p^^] • and 


+Pll 


_ ^ ^ r[pi2.«*So(t+l)+P2ia/bSi(t+l)]^ 

Cr+Piib2sQ (t+l)+P2il/b2Si (t+1) ) 2 

_ b(piiabSo(t+l)+P2ia/bSi(t+l)]|2^ 

r+Piib2So(t+l)+P2il/b2si(t+l) J 


+P. 
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O"-'*,  ■*^21'" 

a _ PiiabSQ(t+l)+P2j^a/bSj^(t+l) 


■I 


Sj^(t+1) 


Sj^(t) 


+p 
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So  (t+1) 


blr+Pij^b2So(t+l)+P2iSi(t+l)/b2,j 
Assuming  k-1  at  time  t,  then  Tr(t+1)  « (pj^2  1*22^  ' 

[r+Pi2b^So (t+1) +P22l/b2Si (t+1) 1 2 
r+Pi2b2So(t+l)+P22l/b^Sf (t+1) 

^ _ Pi2“bSo(t+l)+P22a/bSi(t+l)  ]s^(t+i) 

, b(r+p^2b2Sj,(t+l)+P22S^(t+l)/b2]J 

Note  that  u(t)  switches  from  one  form  to 
another,  depending  on  the  value  of  x(t)  — thus, 
this  solution  depends  on  an  exact  knowledge  of  x(t). 
If  knowledge  of  x(t)  is  corrupted  by  measurement 
noise  (or,  if  u(t)  is  corrupted  by  control  noise), 
then  this  (becomes  a dual  control  problem. 


+P 


22 


To  be  specific,  suppose 
y(t)  « x(t) 

x(t+l)  - ax(t)  + byu(t)  + £(t) 


(16) 


¥(t) 


(1|  x(t)-ax(t-l) 

[oj  u(t-l) 

0..  x(t)-ax(t-l) 
u(t-l) 


(9) 


1/b 


and  k(t)  - 0 if  Tr(t)-[1  0)  ’ or  1 if  l|r(t)-[0  1]  ' 
The  control  u(t)  is  computed  from 

° ” «iu  [ *2*^^^*  tu^ft) 

+ ¥o(t+l)V*(nx(t)+bu(t),k-0,t+l) 

+ ¥i{t+l)V»(ax(t)+^(t),k-l,t+l) 

and  the  assumption  that 


%<here  bj^  and  k are  as  before.  The  control  noise 
^(t)  is  applied  simultaneously  with  u(t)  and  is 
gaussian  %«hlte  noise  with  E[€(t)]wO,  E[C(t)C(T)] 
-56(t-T),  El£(t)x(t))-0,  and  EtC(t)k(t)]-0.  The 
optimal  solution  of  this  problem  requires  order  of 
t2  Riccatl  equations  at  any  time  t because  exact 
)cnowledge  of  ir(t+l)  is  not  given  and  the  control 
affects  the  optimal  estimation  process  for  x(t)  and 
¥ (t) . A suboptimal  controller  would  likely  separate 
the  estimation  function  of  ¥(t)  and  x(t)  from  the 
control  gain  calculation  and  Ignore  the  dual 
control  effect. 


(10) 


If  measurement  noise  is  present t i.e., 

y(t)  - x(t)  + 9(t) 
x(t+l)  ••  ax(t)  + b]^u(t) 


(17) 
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a similar  effect  is  noted. 


4.  Actuator  Failures 


I 

I 


1 


I 

I 

I 

I 


I 


A different  approach  is  taken  in  this  paper; 
here,  it  is  assumed  that  the  controller  has  no 
Information  about  the  current  state  of  the  random 
parameters,  and  that  it  can  obtain  none  through 
observation  of  x (t) . Therefore,  the  control  law 
is  Independent  of  these  parameters  and  x(t) , and 
the  problems  of  dual  control  do  not  arise.  The 
formulation  also  allows  a steady-state  solution 
under  certain  conditions  (which  the  optimal  solu- 
tion in  general  does  not)  which  is  structurally 
identical  to  the  normal  linear  time  invariemt 
optimal  control  law.  This  approach  is  taken 
because  the  complexity  of  the  optimal  solution, 
when  it  exists,  is  too  great  to  allow  a numerical 
comparison  between  systems  and  is  not  constemt, 
but  depend.^  on  where  various  noise  terms  appear  in 
the  system  description.  The  control  solution  pre- 
sented here  is  still  extremely  complex  (k  coupled 
Riccatl  equations,  where  k is  the  number  of  possi- 
ble system  configurations  and  can  be  very  difficult 
to  solve  in  even  the  simplest  of  situations.  The 
information  presented  here  is  in  the  context  of 
reliedjility  and,  more  specifically,  actuator 
failure.  The  concept  is  applicable  to  a much 
broader  remge  of  problems. 


It  is  assumed  in  this  paper  that  each  actuator 
may  fail  to  zero,  and  may  be  repaired  at  a later 
time.  All  failures  are  assumed  to  occur  after  x^(t) 
is  measured  and  before  the  control  input  is  applied. 
Therefore,  if  actuator  ^ fails,  then 

Bp  B j (24) 

where 

Bji  - I ^ I • • • I ^-1  I 0 i ^+1  I ...  I (25) 

Let  B • be  the  set  of  possible  actuator 

configurations.  Assuming  the  probability  of  fail- 
ure of  any  set  of  actuators  at  ttl,  given  that  they 
are  operational  at  t,  is  constant  for  all  t,  then 
the  vector  £(t)  is  defined 

w(t)  e r’'*^  (26) 

irj^(t)  ■ the  prob(d>ility  that  the  actuator 

configuration  is  at  time  t,  (27) 
i-0,...,k. 

If  the  probability  of  failure  of  every  set  of  actua- 
tors is  known,  then  7r(t)  is  propagated  in  time  as 
follows,  assuming  independence  of  n and  x: 


3 . Problem  Formulation 

A linear  quadratic  control  problem  in  discrete 
time  which  accounts  for  system  effectiveness  under 
possible  actuator  failure  is  formulated.  The 
steady-state  solution  to  this  problem,  when  it 
exists,  yields  a direct  con^rison  of  two  linear 
systems.  The  comparison  is  based  on  system  reli- 
ability and  performance. 

This  paper  is  concerned  with  the  discrete 


time  linear  system 

x(t+l)  - to(t)  + Bou(t)  (18) 

where 

x(t)  e r",  u(t)  e and  Ae  R"*  " (19) 

The  matrix  Bp  e R"  * is  assumed  to  be  of  the  form 

Bo  - (bi  I ^ I ^ I . . . I (20) 

with 

biER"  (21) 

corresponding  to  the  control  variable  Uj^  of 

u « [Uj  Uj  U3  . . . Uj„]  ' (22) 


The  system  is  assumed  to  run  over  the  time  set 

T " {-Ti,-Ti+1,-Ti+2,...,T2-2,T2-1,T2} 

Given  any  trajectory  (x(t)  ,u(t)  ),.£^,  the  cost  is 
defined  as 

Tj-l 

J-  J (x*  (t)2x(t)+u’ (t)^(t))+x’ (T2)2fk(T2) 

t— Ti  ~ (23) 


Tr(t+1)  - Pw(t)  (28) 

P “ (Pij)  e r’^+I  X ■'+1  (29) 

p..  = tprob.  of  actuator  configuration  (30) 
^ Bj^  at  time  t+1  | actuator  configura- 
tion Bj  at  time  t} 

The  P matrix  captures  at  a general  level  all 
failure  rates,  repair  rates,  and  reconfiguration 
rates. 


5.  Objective 

We  wish  our  optimization  problem  to  take  into 
account  the  probabilities  of  transfer  between  var- 
ious structural  states  [in  this  case,  actuator 
configurations) . A sufficiently  abstract  framework 
is  found,  by  using  w(t) , so  that  relietbility  can  be 
abstracted  in  the  context  of  a control  problem. 

One  system  cw  then  be  ranked  as  more  reliable  than 
another  from  a control  systems  viewpoint.  We  thus 
understand  failure  by  tying  it  to  the  performance 
index,  and  "more  relieOsle"  is  equated  with  a smaller 
expected  cost. 

This  freunework  involves  open  loop  system 
design,  by  remking  offline  various  system  configura- 
tions, such  as  redundant  sensors  or  functional 
redundancy;  system  failures,  reconfiguration  and 
repair  are  represented  in  the  objective  function. 

In  this  paper  are  presented  the  first  types  of 
problems  which  can  be  handled  using  this  procedure. 


6.  The  Optimization  Problem 

We  wish  to  find  the  closed-loop  non-switching, 
non-dual  feedback  gain  Go  such  that  the  control 
u - Gpx  minimizes  the  expected  cost-to-go,  given 
that  the  actuator  configuration  at  time  -Ti  is 
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r 


i 


i.e.,  that  TH-ly)  = [1  0 0 . . . 0)'. 

We  first  define  Ci(t)u^x 
To-l 

Ci(t)  X “ i U' 

' T-t  ~ 

+ X'  (T7)QfX(T,)|B^  (t)  ,u,x,w(t|x(t))-ir(t)) 

(31) 

where  the  expectation  is  taken  over  all  possible 
sequences  of  actuator  configurations,  given  that 
the  configuration  at  t is  B^Ct),  u euid  x are  the 
sequences  of  control  and  state  vari207les  respec- 
tively, and  the  last  condition  is  the  assumption 
that  iT(t)  is  independent  of  x(t) . Defining  £(t)^^x 
as  the  vector  over  all  i>0,l,...,k  of  these 
expectations,  then  by  theorem  1 (see  Appendix), 

“ P'^C(t+l)  + J(x(t),u(t))  (32) 

where 

J(x,u)  - (x'^t  + u’»i)J.g  (33) 

C(T2)  - (X' (T2)2fX(T2))i.o 

The  expected  cost-to-go  at  time  t,  given  x (t) , u(t) 
and  IT  (t) , is  thus 

<Tt(t),C(t)„  > (35) 

The  problem  is  to  minimize  (35)  w.r.t.  ii(t) . I.e., 

defining  C*(t+l)x  as  the  vector  C(t+1)  resulting 
from  the  application  of  the  minimizing  control 
over  the  time  interval  starting  at  t-fl. 


R + y x.  (t)  y p -b'^s  (t+i)B 
L-  -“JJ 

[ R + S ^(t+l)B^  j 


-1 


-1 


k 

I 

£^0 


(40) 


ii‘^2> 


(41) 


Note  that  u^(t)  is  structurally  like  the  usual 
lineu  quadratic  optimal  control  law;  however,  the 
control  law  takes  account  of  the  different  possible 
actuator  configurations  using  a weighted  sum  of 
forms  containing  the  solutions  to  a set  of  coupled 
Riccatl  equations  (40) . The  control  law  does  not 
account  for  the  information  in  x(t)  about  x(t) , 
and  therefore  does  not  have  switching  gains  which 
are  dependent  on  a function  of  x(t) . 


7.  The  Steady-State  Solution 


<TL(t)  ,C^»(t)x>  - ndn  {<£(t)  ,P^C*(t+l)+J(x,u)>} 
u 136) 


W/  \ W/ 

where 

C*(t+1)  - (C*(t+1)  I ^ + B^u)![_q 
If  we  assume 


c*(t)  - xMt)Si(t)x(t) 


then  we  obtain 


A A 

u»(t)  --[R+  iTri(t)  I Zp,.B;s,(t+l)B,l) 
i-0  j-0 

k k 

• yx.(t)[yp  .B^S^^'(t-t-l)lAx(t) 

£■0  * m-0 


(37) 


(38) 


-1 


(39) 


Si(t) 
k 


(t+1) 


fi. 


.]] 

[fi- 


• B*S^(t+l)  - Sj(t+1) 


X'-LW---"*”--)]' 


We  conjecture  in  this  section  sufficient  condi- 
tions for  the  existence  of  a steady-state  solution. 
The  existence  of  a solution  depends  on  I)  the 
existence  of  a steady-state  value  x,  i.e.,  such 
t)iat  ir^  ■■  P 2L  ' stablllzabllity  of  the 

system  IA,B^],  i-0,...,k.  It  is  interesting  to 
note  that  condition  I is  necessary,  whereas  condi- 
tion II  is  not. 

Conjecture 

(AJ  A steady-state  solution  to  equations  (39)- (41) 
exists  if 

I)  there  exists  a ir  such  that  ir  • pir  and 
£(t)  as  t->«. 

II)  The  systems  (A,B,]  are  stablllzabls  for 

all  i-0,...,k.  ^ 


(B]  X of  part  I exists  iff  one  of  the  following 
three  conditions  is  satisfied  for  each  ele- 
ment of  the  Jordon  normal  form  A of  P, 
where  ~ ~ 


(42) 


(43) 


! 

! 
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For  each  i, 


i) 

I“il  <1 

ii) 

«i  - 1 

iii) 

hil  = 1-  < 

the 

steady-state 

1,1,. 

. .,k  satisfy 

1,  i-  1,  (T  11^) 


tlon  exists,  for  satisfy  the  follow- 

ing set  of  equations. 

Si-  joPjilA'S^AtG'B'  S^B.G-G-B-  S^A 
■ A'S  j B j G)  + £ + £'R£  <S1> 

9.  Conclusion 


• ( •«)  ‘i^Sj 

• (•!£,£)  *«](«♦  X',£Ui£. 


-1 


-1 


H-O 
A + £ 


(44) 


The  corresponding  steady-state  control  is 
I'  T 1 T 

U*--(R+  I IfiBiS  B J ITj^B^S  )Ax  (45) 

i»0  i“0 

The  steady-state  cost-to-go  at  time  t,  given  ^(t) , 
is 


This  paper  reports  on  a class  of  linear  quadra- 
tic control  problems  in  discrete-time,  where  actua- 
tor failure  may  occur.  The  major  contribution  of 
this  work  is  that  a steady-state,  structure-indepen- 
dent control  law  can  be  defined,  along  with  its 
associated  cost.  This  allows  a numerical  comparison 
of  different  control  methodologies  to  be  performed. 

The  case  for  a suJsoptimal  solution  to  be  used 
for  comparisons  is  made  by  the  decrease  in  complex- 
ity over  euiy  optimal  solution  which  might  exist. 
Although  this  solution  is  still  quite  complex,  it 
is  solveUsle  and  easier  conceptually  than  the  dual 
control  problem,  whose  solution  has  little  applica- 
tion due  to  the  increased  complexity  and  non- 
existence of  a steady-state  solution. 

The  concept  behind  this  paper  is  much  broader 
in  scope  than  is  shown  here.  The  special  structure 
of  the  B-matrix  allows  one  to  tie  together  the 
concepts  of  stabilizeUsllity  of  individual  structures 
and  of  existence  of  a solution.  This  cannot  t>e 
done  in  general.  There  are  very  simple  exan^les 
that  clearly  demonstrate,  for  Instance,  the 
uncertainty  threshold  principle. 


10 . Appendix 


<]L.C(x(t))> 


(46) 


Theorem  1; 


where 

C(x(t))  - (*^(t)S^x(t))^_Q  (47) 

8.  A Method  for  Comparing  Two  Lineeu: 
Feedbac)c  Control  Laws 

In  the  last  section,  the  equations  were  given 
for  the  steady-state  optimal  cost-to-go  as  a func- 
tion of  X.  In  this  section,  these  equations  are 
extended  for  the  case  of  the  control  law 

u - - Gx  (48) 

The  solution  of  these  equations  yields  a valid 
comparison  between  two  linear  feedjMcIc  control  laws 
when  it  exists.  The  coat  of  control  u ~ -Gx  la 
defined  as  a function  of  x 

V(G)  - <!,.£, (G,x)>  (49) 

whore  C now  depends  on  G,  as  well  as  x.  Assuming  C 
is  of  the  form 

C(G,x)  ■ I«'Sd(0)x|  . . . I i'Sjj(G)xl^  (50) 
whore  Sj.  depends  on  G,  than  if  s steady  state  solu- 


For  the  system  given  in  section  (3),  if  Cj^(t)  is 
the  cost-to-go,  given  the  structure  i (B-matrix  » 
B^)  as  a function  of  { (x(t) ,u(£) )^^^  ,x(T2)},  then 
Cj^  (t)  can  be  computed  recursively  as 


k 

Ci(t)  - ^ probability (i -►  j at  t)c . (t+l)+j  (t) 

j-0  ^ ,(A.l) 

where 


and. 


x' 

(t)ax(t)  +u' 

(t)R u(t)  I 

tXTj 

j(t) 

(x,u) 

■ 

X' 

(T2)Qfx(T,) 

; t - Tj 

(A. 2) 

if  C 

(t)  - 

L(t) 

^\.o'  ■ 

ft 

o 

, then 

C(t) 

Oil 

1 

C(t+1) 

+ J(t) 

(A.3) 

Proof  I 

cj(t)  is  the  expectation  of  J({(x()l.)  ,u(t))  j2^^x(T^)) 

whore  J(')  ■ J j(t)(*)  as  in  equation  (8),  given 
tex 

that  the  structure  is  i at  time  t.  Since  this 
expectation  is  a sum  of  susis,  and  since  the  space 
of  sequences  {i,  e {0,1, . . . ,)c}}  ^ *• 

then  Pubini's  theorem  applies 
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(A. 4) 


c^(t)  - (cost  over  rt,t+l)J 
k 

+ J prob. (structure  j at  time  t+1) 
j-0  • [cost-to-go  at  time  t+1, 
given  structure  jj 

which  is  exactly  (A.l).  Q.E.D. 
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